Graphical user interface for a surgical navigation system and method for providing an augmented reality image during operation

ABSTRACT

A surgical navigation system includes: a 3D display system with a see-through visor; a tracking system comprising means for real-time tracking of: a surgeon&#39;s head, the see-through visor, a patient anatomy and a surgical instrument to provide current position and orientation data; a source of an operative plan, a patient anatomy data and a virtual surgical instrument model; a surgical navigation image generator configured to generate a surgical navigation image with a three-dimensional image representing simultaneously a virtual image of the surgical instrument corresponding to the current position and orientation of the surgical instrument and a virtual image of the surgical instrument indicating the suggested positions and orientation of the surgical instrument according to the operative plan data based on the current relative position and orientation of the surgeon&#39;s head, the see-through visor, the patient anatomy and the surgical instrument; wherein the 3D display system is configured to show the surgical navigation image at the see-through visor, such that an augmented reality image collocated with the patient anatomy in the surgical field underneath the see-through visor is visible to a viewer looking from above the see-through visor towards the surgical field.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/842,793, filed Apr. 8, 2020, entitled “GRAPHICAL USER INTERFACE FOR ASURGICAL NAVIGATION SYSTEM AND METHOD FOR PROVIDING AN AUGMENTED REALITYIMAGE DURING OPERATION,” which is a continuation of U.S. patentapplication Ser. No. 16/059,061, filed Aug. 9, 2018, entitled “GRAPHICALUSER INTERFACE FOR A SURGICAL NAVIGATION SYSTEM AND METHOD FOR PROVIDINGAN AUGMENTED REALITY IMAGE DURING OPERATION,” now U.S. Pat. No.10,646,285, which claims priority under 35 U.S.C. § 119 to the EuropeanPatent Application No. 17186307, filed Aug. 15, 2017, entitled “AGRAPHICAL USER INTERFACE FOR A SURGICAL NAVIGATION SYSTEM FOR PROVIDINGAN AUGMENTED REALITY IMAGE DURING OPERATION,” the disclosures of each ofwhich is hereby incorporated herein by reference in its entirety.

TECHNICAL FIELD

The present disclosure relates to graphical user interfaces for surgicalnavigation systems, in particular to a system and method for operativeplanning and execution of a medical procedure.

BACKGROUND

Some of typical functions of a computer-assisted surgery (CAS) systemwith navigation include presurgical planning of a procedure andpresenting preoperative diagnostic information and images in usefulformats. The CAS system presents status information about a procedure asit takes place in real time, displaying the preoperative plan along withintraoperative data. The CAS system may be used for procedures intraditional operating rooms, interventional radiology suites, mobileoperating rooms or outpatient clinics. The procedure may be any medicalprocedure, whether surgical or non-surgical.

Surgical navigation systems are used to display the position andorientation of surgical instruments and medical implants with respect topresurgical or intraoperative medical imagery datasets of a patient.These images include pre and intraoperative images, such astwo-dimensional (2D) fluoroscopic images and three-dimensional (3D)magnetic resonance imaging (MM) or computed tomography (CT).

Navigation systems locate markers attached or fixed to an object, suchas surgical instruments and patient. Most commonly these trackingsystems are optical and electro-magnetic. Optical tracking systems haveone or more stationary cameras that observes passive reflective markersor active infrared LEDs attached to the tracked instruments or thepatient. Eye-tracking solutions are specialized optical tracking systemsthat measure gaze and eye motion relative to a user's head.Electro-magnetic systems have a stationary field generator that emits anelectromagnetic field that is sensed by coils integrated into trackedmedical tools and surgical instruments.

SUMMARY OF THE INVENTION

Incorporating image segmentation processes that automatically identifyvarious bone landmarks, based on their density, can increase planningaccuracy. One such bone landmark is the spinal pedicle, which is made upof dense cortical bone making its identification utilizing imagesegmentation easier. The pedicle is used as an anchor point for varioustypes of medical implants. Achieving proper implant placement in thepedicle is heavily dependent on the trajectory selected for implantplacement. Ideal trajectory is identified by surgeon based on review ofadvanced imaging (e.g., CT or MRI), goals of the surgical procedure,bone density, presence or absence of deformity, anomaly, prior surgery,and other factors. The surgeon then selects the appropriate trajectoryfor each spinal level. Proper trajectory generally involves placing anappropriately sized implant in the center of a pedicle. Idealtrajectories are also critical for placement of inter-vertebralbiomechanical devices.

Another example is placement of electrodes in the thalamus for thetreatment of functional disorders, such as Parkinson's. The mostimportant determinant of success in patients undergoing deep brainstimulation surgery is the optimal placement of the electrode. Propertrajectory is defined based on preoperative imaging (such as Mill or CT)and allows for proper electrode positioning.

Another example is minimally invasive replacement of prosthetic/biologicmitral valve in for the treatment of mitral valve disorders, such asmitral valve stenosis or regurgitation. The most important determinantof success in patients undergoing minimally invasive mitral valvesurgery is the optimal placement of the three dimensional valve.

The fundamental limitation of surgical navigation systems is that theyprovide restricted means of communicating to the surgeon.Currently-available navigation systems present some drawbacks.

Typically, one or several computer monitors are placed at some distanceaway from the surgical field. They require the surgeon to focus thevisual attention away from the surgical field to see the monitors acrossthe operating room. This results in a disruption of surgical workflow.Moreover, the monitors of current navigation systems are limited todisplaying multiple slices through three-dimensional diagnostic imagedatasets, which are difficult to interpret for complex 3D anatomy.

The fact that the screen of the surgical navigation system is locatedaway from the region of interest (ROI) of the surgical field requiresthe surgeon to continuously look back and forth between the screen andthe ROI. This task is not intuitive and results in a disruption tosurgical workflow and decreases planning accuracy.

For example, a system of such type is disclosed in a U.S. Pat. No.9,532,848, which discloses a system for assisting a user manipulating anobject during a surgery, the system comprising a tracking device fortracking the object and for generating tracking data for the object anda sterilized displaying device located in a volume within the sterilefield defined as being above a plane of and delimited by an operatingtable and below the shoulders of an operator standing next to a patientlying on the operating table, the displaying device being supporteddirectly by the operating table. Even though the displaying device ispositioned very close to the patient being operated, the surgeon stillneeds to look back and forth between the screen and the ROI.

When defining and later executing an operative plan, the surgeoninteracts with the navigation system via a keyboard and mouse,touchscreen, voice commands, control pendant, foot pedals, hapticdevices, and tracked surgical instruments. Based on the complexity ofthe 3D anatomy, it can be difficult to simultaneously position andorient the instrument in the 3D surgical field only based on theinformation displayed on the monitors of the navigation system.Similarly, when aligning a tracked instrument with an operative plan, itis difficult to control the 3D position and orientation of theinstrument with respect to the patient anatomy. This can result in anunacceptable degree of error in the preoperative plan that willtranslate to poor surgical outcome. There is disclosed a surgicalnavigation system comprising: a 3D display system with a see-throughvisor; a tracking system comprising means for real-time tracking of: asurgeon's head, the see-through visor, a patient anatomy and a surgicalinstrument to provide current position and orientation data; a source ofan operative plan, a patient anatomy data and a virtual surgicalinstrument model; a surgical navigation image generator configured togenerate a surgical navigation image comprising a three-dimensionalimage representing simultaneously a virtual image of the surgicalinstrument corresponding to the current position and orientation of thesurgical instrument and a virtual image of the surgical instrumentindicating the suggested positions and orientation of the surgicalinstrument according to the operative plan data based on the currentrelative position and orientation of the surgeon's head, the see-throughvisor, the patient anatomy and the surgical instrument; wherein the 3Ddisplay system is configured to show the surgical navigation image atthe see-through visor, such that an augmented reality image collocatedwith the patient anatomy in the surgical field underneath thesee-through visor is visible to a viewer looking from above thesee-through visor towards the surgical field.

The three-dimensional image of the surgical navigation image may furthercomprise at least one of: the patient anatomy data, operative plan data,in accordance to the current position and orientation data provided bythe tracking system.

The three-dimensional image of the surgical navigation image may furthercomprise a graphical cue indicating the required change of position andorientation of the surgical instrument to match the suggested positionand orientation according to the pre-operative plan data.

The surgical navigation image may further comprise a set of orthogonal(axial, sagittal, and coronal) and arbitrary planes of the patientanatomy data.

The 3D display system may comprise a 3D projector and a see-throughprojection screen, wherein the 3D projector is configured to project thesurgical navigation image onto the see-through projection screen, whichis partially transparent and partially reflective, for showing thesurgical navigation image.

The 3D display system may comprise a 3D projector, an opaque projectionscreen and a see-through mirror, wherein the 3D projector is configuredto project the surgical navigation image onto the opaque projectionscreen for showing the surgical navigation image for emission towardsthe see-through mirror, which is partially transparent and partiallyreflective.

The 3D display system may comprise a 3D projector, a plurality of opaquemirrors, an opaque projection screen and a see-through mirror, whereinthe 3D projector is configured to project the surgical navigation imagetowards the plurality of opaque mirrors for reflecting the surgicalnavigation image towards the opaque projection screen for showing thesurgical navigation image for emission towards the see-through mirror,which is partially transparent and partially reflective.

The 3D display may comprise a 3D monitor for showing the surgicalnavigation image for emission towards the see-through mirror which ispartially transparent and partially reflective.

The 3D display may comprise a see-through 3D screen, which is partiallytransparent and partially emissive, for showing the surgical navigationimage.

The see-through visor may be configured to be positioned, when thesystem is in use, at a distance (d1) from the surgeon's head which isshorter than the distance (d2) from the surgical field of the patientanatomy.

The surgical navigation image generator may be controllable by an inputinterface comprising at least one of: foot-operable pedals, amicrophone, a joystick, an eye-tracker.

The tracking system may comprise a plurality of arranged fiducialmarkers, including a head array, a display array, a patient anatomyarray, an instrument array; and a fiducial marker tracker configured todetermine in real time the positions and orientations of each of thecomponents of the surgical navigation system.

At least one of the head array, the display array, the patient anatomyarray, the instrument array may contain several fiducial markers thatare not all coplanar.

There is also disclosed a method for providing an augmented realityimage during an operation, comprising: providing a 3D display systemwith a see-through visor; providing a tracking system comprising meansfor real-time tracking of: a surgeon's head, the 3D see-through visor, apatient anatomy and a surgical instrument to provide current positionand orientation data; providing a source of: an operative plan, apatient anatomy data and a virtual surgical instrument model;generating, by a surgical navigation image generator, a surgicalnavigation image comprising: a three-dimensional image representingsimultaneously a virtual image of the surgical instrument correspondingto the current position and orientation of the surgical instrument and avirtual image of the surgical instrument indicating the suggestedpositions and orientations of the surgical instruments according to theoperative plan based on the current relative position and orientation ofthe surgeon's head, the see-through visor, the patient anatomy and thesurgical instrument; showing the surgical navigation image at thesee-through visor, such that an augmented reality image collocated withthe patient anatomy in the surgical field underneath the see-throughvisor is visible to a viewer looking from above the see-through visortowards the surgical field.

The intended use of this invention is both presurgical planning of idealsurgical instrument trajectory and placement, and intraoperativesurgical guidance, with the objective of helping to improve surgicaloutcomes.

A combination of a navigated probe and a computer-assisted medicalsystem is used for interactive creation of a trajectory for positioningof a medical device. A navigated probe facilitates the positioning ofthe medical device. The navigated probe is part of a computer-assistedmedical system consisting of a 6-degree-of-freedom (DOF) tracker(optical, electromagnetic, inertial, or any other tracking technology)and a navigated structure. The navigated structure contains a graphicaluser interface (GUI) for displaying patient anatomy in three dimensions(3D), as well as a virtual representation of actual implanted medicaldevices (IMDs) and instruments during a surgical procedure in a realtime.

The surgeon can control the navigated probe by looking at its virtualrepresentation on the GUI and lining it up to the virtual representationof the organ to achieve a proper trajectory for medical implantplacement. During the planning process, a virtual instrument isdisplayed on a 3D display device to indicate the dynamic 3D position andorientation of the medical device. The surgeon can interact with theprobe and the computer-assisted medical system by either using a 6-DOFtracking device and/or by pressing on a set of pre-programed pedals orusing other input interfaces, such as a microphone (for voice commands),a joystick, an eye-tracker (for gaze tracking).

The presented system and method solve the critical problems of typicalsurgical navigation systems. First, they allow the surgeon to focus thevisual attention to the surgical field by superimposing 3D patientanatomy, surgical guidance, and orthogonal planes directly onto the areaof patient anatomy where the surgery is performed, without requiring thesurgeon to look away from the surgical field. Secondly, they provide thesurgeon a more intuitive mechanism to define and execute an operativeplan by simply handling the surgical instruments in a 3D workspace thatperfectly matches the operative field, without requiring the surgeon toperform a disorienting mental mapping of the information displayed bythe navigation system to the 3D position and orientation of the surgicalinstruments with respect to the complex 3D anatomy. Moreover, by usingthe presented system and method, the time of operation can be reduced,as the more intuitive communication means do not distract the surgeonand do not require additional time to look away from the ROI.

These and other features, aspects and advantages of the invention willbecome better understood with reference to the following drawings,descriptions and claims.

BRIEF DESCRIPTION OF FIGURES

The surgical navigation system and method are presented herein by meansof non-limiting example embodiments shown in a drawing, wherein:

FIG. 1A shows a layout of a surgical room employing the surgicalnavigation system in accordance with an embodiment of the invention;

FIG. 1B shows a layout of a surgical room employing the surgicalnavigation system in accordance with an embodiment of the invention;

FIG. 1C shows a layout of a surgical room employing the surgicalnavigation system in accordance with an embodiment of the invention;

FIG. 2A shows components of the surgical navigation system in accordancewith an embodiment of the invention;

FIG. 2B shows components of the surgical navigation system in accordancewith an embodiment of the invention;

FIG. 3A shows an example of an augmented reality display in accordancewith an embodiment of the invention;

FIG. 3B shows an example of an augmented reality display in accordancewith an embodiment of the invention;

FIG. 3C shows an example of an augmented reality display in accordancewith an embodiment of the invention;

FIG. 3D shows an example of an augmented reality display in accordancewith an embodiment of the invention;

FIG. 3E shows an example of an augmented reality display in accordancewith an embodiment of the invention;

FIG. 4A shows an embodiment of a 3D display system.

FIG. 4B shows another embodiment of a 3D display system.

FIG. 4C shows another embodiment of a 3D display system.

FIG. 4D shows another embodiment of a 3D display system.

FIG. 4E shows another embodiment of a 3D display system.

FIG. 5A show eye tracking in accordance with an embodiment of theinvention.

FIG. 5B show eye tracking in accordance with an embodiment of theinvention.

DETAILED DESCRIPTION

The following detailed description is of the best currently contemplatedmodes of carrying out the invention. The description is not to be takenin a limiting sense, but is made merely for the purpose of illustratingthe general principles of the invention.

The system presented herein is comprises a 3D display system 140 to beimplemented directly on real surgical applications in a surgical room asshown in FIGS. 1A-1C. The 3D display system 140 as shown in the exampleembodiment comprises a 3D display 142 for emitting a surgical navigationimage 142A towards a see-through mirror 141 that is partiallytransparent and partially reflective, such that an augmented realityimage 141A collocated with the patient anatomy in the surgical field 108underneath the see-through mirror 141 is visible to a viewer lookingfrom above the see-through mirror 141 towards the surgical field 108.

The surgical room typically comprises a floor 101 on which an operatingtable 104 is positioned. A patient 105 lies on the operating table 104while being operated by a surgeon 106 with the use of various surgicalinstruments 107. The surgical navigation system as described in detailsbelow can have its components, in particular the 3D display system 140,mounted to a ceiling 102, or alternatively to the floor 101 or a sidewall 103 of the operating room. Furthermore, the components, inparticular the 3D display system 140, can be mounted to an adjustableand/or movable floor-supported structure (such as a tripod). Componentsother than the 3D display system 140, such as the surgical imagegenerator 131, can be implemented in a dedicated computing device 109,such as a stand-alone PC computer, which may have its own inputcontrollers and display(s) 110.

In general, the system is designed for use in such a configurationwherein the distance d1 between the surgeon's eyes and the see-throughmirror 141, is shorter than the distance d2, between the see-throughmirror 141 and the operative field at the patient anatomy 105 beingoperated.

FIG. 2A shows a functional schematic presenting connections between thecomponents of the surgical navigation system and FIG. 2B shows examplesof physical embodiments of various components.

The surgical navigation system comprises a tracking system for trackingin real time the position and/or orientation of various entities toprovide current position and/or orientation data. For example, thesystem may comprise a plurality of arranged fiducial markers, which aretrackable by a fiducial marker tracker 125. Any known type of trackingsystem can be used, for example in case of a marker tracking system,4-point marker arrays are tracked by a three-camera sensor to providemovement along six degrees of freedom. A head position marker array 121can be attached to the surgeon's head for tracking of the position andorientation of the surgeon and the direction of gaze of the surgeon—forexample, the head position marker array 121 can be integrated with thewearable 3D glasses 151 or can be attached to a strip worn oversurgeon's head.

A display marker array 122 can be attached to the see-through mirror 141of the 3D display system 140 for tracking its position and orientation,as the see-through mirror 141 is movable and can be placed according tothe current needs of the operative setup.

A patient anatomy marker array 123 can be attached at a particularposition and orientation of the anatomy of the patient.

A surgical instrument marker array 124 can be attached to the instrumentwhose position and orientation shall be tracked.

Preferably, the markers in at least one of the marker arrays 121-124 arenot coplanar, which helps to improve the accuracy of the trackingsystem.

Therefore, the tracking system comprises means for real-time tracking ofthe position and orientation of at least one of: a surgeon's head 106, a3D display 142, a patient anatomy 105, and surgical instruments 107.Preferably, all of these elements are tracked by a fiducial markertracker 125.

A surgical navigation image generator 131 is configured to generate animage to be viewed via the see-through mirror 141 of the 3D displaysystem. It generates a surgical navigation image 142A comprising datarepresenting simultaneously a virtual image 164B of the surgicalinstrument corresponding to the current position and orientation of thesurgical instrument and a virtual image 164A of the surgical instrumentindicating the suggested positions and orientation of the surgicalinstrument according to the operative plan data 161, 162 based on thecurrent relative position and orientation of the surgeon's head 106, thesee-through visor 141, 141B, 141D, the patient anatomy 105 and thesurgical instrument 107. It may further comprise data representing thepatient anatomy scan 163 (which can be generated before the operation orlive during the operation).

The surgical navigation image generator 131, as well as other componentsof the system, can be controlled by a user (i.e. a surgeon or supportstaff) by one or more user interfaces 132, such as foot-operable pedals(which are convenient to be operated by the surgeon), a keyboard, amouse, a joystick, a button, a switch, an audio interface (such as amicrophone), a gesture interface, a gaze detecting interface etc. Theinput interface(s) are for inputting instructions and/or commands.

All system components are controlled by one or more computer which iscontrolled by an operating system and one or more software applications.The computer may be equipped with a suitable memory which may storecomputer program or programs executed by the computer in order toexecute steps of the methods utilized in the system. Computer programsare preferably stored on a non-transitory medium. An example of anon-transitory medium is a non-volatile memory, for example a flashmemory while an example of a volatile memory is RAM. The computerinstructions are executed by a processor. These memories are exemplaryrecording media for storing computer programs comprisingcomputer-executable instructions performing all the steps of thecomputer-implemented method according the technical concept presentedherein. The computer(s) can be placed within the operating room oroutside the operating room. Communication between the computers and thecomponents of the system may be performed by wire or wirelessly,according to known communication means.

The aim of the system is to generate, via the see-through visor 141, anaugmented reality image such as shown in examples of FIGS. 3A-3E. Whenthe surgeon looks via the see-through visor 141, the surgeon sees theaugmented reality image 141A which comprises:

-   -   the real world image: the patient anatomy, surgeon's hands and        the instrument currently in use (which may be partially inserted        into the patient's body and hidden under the skin);    -   and a computer-generated surgical navigation image 142A        comprising:        -   a 3D image 171 representing at least one of: the virtual            image of the patient anatomy 163, the virtual image of the            instrument 164 or surgical guidance indicating suggested            (ideal) trajectory and placement of surgical instruments            107, according to the pre-operative plans 161 (as shown in            FIG. 3C);        -   preferably, three different orthogonal planes of the patient            anatomy data 163: coronal 174, sagittal 173, axial 172;        -   preferably, a menu 175 for controlling the system operation.

If the 3D display 142 is stereoscopic, the surgeon shall use a pair of3D glasses 151 to view the augmented reality image 141A. However, if the3D display 142 is autostereoscopic, it may be not necessary for thesurgeon to use the 3D glasses 151 to view the augmented reality image141A.

Preferably, the images of the orthogonal planes 172, 173, 174 aredisplayed in an area next (preferably, above) to the area of the 3Dimage 171, as shown in FIG. 3A, wherein the 3D image 171 occupies morethan 50% of the area of the see-through visor 141.

The location of the images of the orthogonal planes 172, 173, 174 may beadjusted in real time depending on the location of the 3D image 171,when the surgeon changes the position of the head during operation, suchas not to interfere with the 3D image 171.

Therefore, in general, the anatomical information of the user is shownin two different layouts that merge for an augmented and mixed realityfeature. The first layout is the anatomical information that isprojected in 3D in the surgical field. The second layout is in theorthogonal planes.

The surgical navigation image 142A is generated by the image generator131 in accordance with the tracking data provided by the fiducial markertracker 125, in order to superimpose the anatomy images and theinstrument images exactly over the real objects, in accordance with theposition and orientation of the surgeon's head. The markers are trackedin real time and the image is generated in real time. Therefore, thesurgical navigation image generator 131 provides graphics rendering ofthe virtual objects (patient anatomy, surgical plan and instruments)collocated to the real objects according to the perspective of thesurgeon's perspective.

For example, surgical guidance may relate to suggestions (virtualguidance clues 164) for placement of a pedicle screw in spine surgery orthe ideal orientation of an acetabular component in hip arthroplastysurgery. These suggestions may take a form of animations that show thesurgeon whether the placement is correct. The suggestions may bedisplayed both on the 3D holographic display and the orthogonal planes.The surgeon may use the system to plan these orientations before orduring the surgical procedure.

In particular, the 3D image 171 is adapted in real time to the positionand orientation of the surgeon's head. The display of the differentorthogonal planes 172, 173, 174 may be adapted according to the currentposition and orientation of the surgical instruments used.

The aligning the line of sight of the surgeon onto the see-throughmirror with the patient anatomy underneath the see-through mirror,involving the scaling and orientation of the image, can be realizedbased on known solutions in the field of computer graphics processing,in particular for virtual reality, including virtual scene generation,using well-known mathematical formulas and algorithms related to viewercentered perspective. For example, such solutions are known from varioustutorials and textbooks (such as “The Future of the CAVE” by T. A.DeFanti et al, Central European Journal of Engineering, 2010, DOI:10.2478/s13531-010-0002-5).

FIG. 3B shows an example indicating collocation of the virtual image ofthe patient anatomy 163 and the real anatomy 105.

For example, as shown in FIG. 3C, the 3D image 171 may demonstrate amismatch between a supposed/suggested position of the instrumentaccording to the pre-operative plan 161, displayed as a first virtualimage of the instrument 164A located at its supposed/suggested position,and an actual position of the instrument, visible either as the realinstrument via the see-through display and/or a second virtual image ofthe instrument 164B overlaid on the current position of the instrument.Additionally, graphical guiding cues, such as arrows 165 indicating thedirection of the supposed change of position, can be displayed.

FIG. 3D shows a situation wherein the tip of the supposed position ofthe instrument displayed as the first virtual image 164A according tothe pre-operative plan 161 matches the tip of the real surgicalinstrument visible or displayed as the second virtual image 164B.However, the remainder of objects do not match, therefore the graphicalcues 165 still indicate the need to change position. The surgicalinstrument is close to the correct position and the system may provideinformation on how close the surgical instrument is to the plannedposition.

FIG. 3E shows a situation wherein the supposed position of the realsurgical instrument matches the position of the instrument according tothe pre-operative plan 161, i.e. the correct position for surgery. Inthis situation the graphical cues 165 are no longer displayed, but thevirtual images 164A, 164B may be changed to indicate the correctposition, e.g. by highlighting it or blinking.

The see-through mirror (also called a half-silvered mirror) 141 is atleast partially transparent and partially reflective, such that theviewer can see the real world behind the mirror but the mirror alsoreflects the surgical navigation image generated by the displayapparatus located above it.

For example, a see-through mirror as commonly used in teleprompters canbe used. For example, the see-through mirror 141 can have a reflectiveand transparent rate of 50R/50T, but other rates can be used as well.

The surgical navigation image is emitted from above the see-throughmirror 141 by the 3D display 142.

In an example embodiment as shown in FIGS. 4A and 4B, a special designof the 3D display 142 is provided that is compact in size to facilitateits mounting within a limited space at the operating room. That designallows generating images of relatively large size, taking into accountthe small distance between the 3D display 142 and the see-through mirror141, without the need to use wide-angle lens that could distort theimage.

The 3D display 142 comprises a 3D projector 143, such as a DLPprojector, that is configured to generate an image, as shown in FIG. 4B(by the dashed lines showing image projection and solid lines showingimages generated on particular reflective planes). The image from the 3Dprojector 143 is firstly refracted by an opaque top mirror 144, then itis refracted by an opaque vertical mirror 145 and subsequently placed onthe correct dimensions on a projection screen 146 (which can be simply aglass panel). The projection screen 146 works as a rear-projectionscreen or a small bright 3D display. The image displayed at theprojection screen 146 is reflected by the see-through mirror 141 whichworks as an augmented reality visor. Such configuration of the mirrors144, 145 allows the image generated by the 3D projector 143 to be shownwith an appropriate size at the projection screen 146. The fact that theprojection screen 146 emits an enlarged image generated by the 3Dprojector 143 makes the emitted surgical navigation image bright, andtherefore well visible when reflected at the see-through mirror 141.Reference 141A indicates the augmented reality image as perceived by thesurgeon when looking at the see-through mirror 141.

The see-through mirror 141 is held at a predefined position with respectto the 3D projector 143, in particular with respect to the 3D projector143, by an arm 147, which may have a first portion 147A fixed to thecasing of the 3D display 142 and a second portion 147B detachably fixedto the first portion 147A. The first portion 147A may have a protectivesleeve overlaid on it. The second portion 147B, together with thesee-through mirror 141, may be disposable in order to keep sterility ofthe operating room, as it is relatively close to the operating field andmay be contaminated during the operation. The arm can also be foldableupwards to leave free space of the work space when the arm and augmentedreality are not needed.

In alternative embodiments, as shown for example in FIGS. 4C, 4D, 4E,alternative devices may be used in the 3D display system 140 in place ofthe see-through mirror 141 and the 3D display 142.

As shown in FIG. 4C, a 3D monitor 146A can be used directly in place ofthe projection screen 146.

As shown in FIG. 4D, a 3D projector 143 can be used instead of the 3Ddisplay 142 of FIG. 4A, to project the surgical navigation image onto asee-through projection screen 141B, which is partially transparent andpartially reflective, for showing the surgical navigation image 142A andallowing to see the surgical field 108. A lens 141C can be used toprovide appropriate focal position of the surgical navigation image.

As shown in FIG. 4E, the surgical navigation image can be displayed at athree-dimensional see-through screen 141D and viewed by the user via alens 141C used to provide appropriate focal position of the surgicalnavigation image.

Therefore, see-through screen 141B, the see-through display 141D and thesee-through mirror 141 can be commonly called a see-through visor.

If a need arises to adapt the position of the augmented reality screenwith respect to the surgeon's head (for example, to accommodate theposition depending on the height of the particular surgeon), theposition of the whole 3D display system 140 can be changed, for exampleby manipulating an adjustable holder (a surgical boom) 149 on FIG. 1A,by which the 3D display 142 is attachable to an operating roomstructure, such as a ceiling, a wall or a floor.

An eye tracker 148 module can be installed at the casing of the 3Ddisplay 142 or at the see-through visor 141 or at the wearable glasses151, to track the position and orientation of the eyes of the surgeonand input that as commands via the gaze input interface to control thedisplay parameters at the surgical navigation image generator 131, forexample to activate different functions based on the location that isbeing looked at, as shown in FIGS. 5A and 5B.

For example, the eye tracker 148 may use infrared light to illuminatethe eyes of the user without affecting the visibility of the user,wherein the reflection and refraction of the patterns on the eyes areutilized to determine the gaze vector (i.e. the direction at which theeye is pointing out). The gaze vector along with the position andorientation of the user's head is used to interact with the graphicaluser interface. However, other eye tracking algorithms techniques can beused as well.

It is particularly useful to use the eye tracker 148 along with thepedals 132 as the input interface, wherein the surgeon may navigate thesystem by moving a cursor by eye sight and inputting commands (such asselect or cancel) by pedals.

While the invention has been described with respect to a limited numberof embodiments, it will be appreciated that many variations,modifications and other applications of the invention may be made.Therefore, the claimed invention as recited in the claims that follow isnot limited to the embodiments described herein.

1.-14. (canceled)
 15. An apparatus, comprising: a memory; a processoroperatively coupled to the memory, the processor configured to:determine, based on data associated with an operative plan of a surgicalprocedure, a suggested position and orientation of a medical device inan anatomy of a patient, the medical device configured to be used in thesurgical procedure; determine, based on tracking data associated withthe medical device, an actual position and orientation of the medicaldevice; and generate a surgical navigation image includingthree-dimensional (3D) image including (1) a virtual object indicativeof the suggested position and orientation of the medical device, (2) avirtual representation of the medical device, and (3) a virtualrepresentation of a portion of the anatomy of the patient; and a displaysystem configured to display the surgical navigation image on a surfacepositionable between a head of an operator and a surgical fieldincluding the patient anatomy such that the virtual representation ofthe portion of the anatomy is collocated with the anatomy and thevirtual representation of the medical device is overlaid on a portion ofthe medical device in a field of view of the operator.
 16. The apparatusof claim 15, wherein the virtual representation of the medical device isa first virtual representation of the medical device, and the virtualobject is a second virtual representation of the medical device havingthe suggested position and orientation of the medical device.
 17. Theapparatus of claim 15, wherein the processor is configured to generatethe surgical navigation image to further include a set of imagesdepicting different orthogonal or arbitrary planes of the portion of theanatomy.
 18. The apparatus of claim 17, wherein the display system isconfigured to display the surgical navigation image on the surface suchthat the set of images is displayed at a location next to the 3D image.19. The apparatus of claim 17, wherein the display system is configuredto adjust the location of the set of images based on a location of the3D image.
 20. The apparatus of claim 17, wherein the display system isconfigured to display the surgical navigation image on the surface suchthat the 3D image occupies a larger area of the field of view of theoperator than the set of images.
 21. The apparatus of claim 17, whereinthe processor is further configured to adapt the set of images of thesurgical navigation image based on changes to the actual position andorientation of the medical device.
 22. The apparatus of claim 15,wherein the processor is further configured to adapt the 3D image basedon a position and orientation of the head of the operator.
 23. Theapparatus of claim 15, wherein the processor is further configured togenerate the surgical navigation image to include virtual guidanceindicating whether a placement of the medical device is correct based onthe suggested position and orientation of the medical device.
 24. Theapparatus of claim 23, wherein the virtual guidance includes ananimation.
 25. The apparatus of claim 15, wherein the surface is asurface of a see-through mirror, the see-through mirror positionable afirst distance from the head of the operator and a second distance fromthe surgical field, the first distance being less than the seconddistance.
 26. The apparatus of claim 25, wherein the display system isconfigured to move such that a location of the see-through mirror canmove relative to the head of the operator.
 27. The apparatus of claim25, wherein the display system includes a 3D display and an armextending from the 3D display, the arm configured to support thesee-through mirror such that the see-through mirror can be positionedbetween the head of the operator and the surgical field when the 3Ddisplay is positioned above the head of the operator.
 28. The apparatusof claim 15, wherein the processor is further configured to: determine,based on tracking data associated with the head of the operator, aposition and orientation of the head of the operator, the processorconfigured to generate the surgical navigation image according to aperspective of the operator based on the position and orientation of thehead of the operator.
 29. A method, comprising: determining, based ondata associated with an operative plan of a surgical procedure, asuggested position and a suggested orientation of a medical device in ananatomy of a patient, the medical device configured to be used in thesurgical procedure; receiving, from a tracking system, tracking dataassociated with the medical device; determining, based on the trackingdata associated with the medical device, an actual position andorientation of the medical device; generating a three-dimensional (3D)image including (1) a first virtual guidance clue for indicating thesuggested position of the medical device, (2) a second virtual guidanceclue for indicating the suggested orientation of the medical device, and(3) a virtual representation of the medical device associated with theactual position and orientation of the medical device; and displayingthe 3D image in a field of view of an operator such that the virtualrepresentation of the medical device is overlaid on a portion of themedical device in the field of view and the first and second virtualguidance clues show the operator whether the medical device has beenplaced in the suggested position and the suggested orientation.
 30. Themethod of claim 29, wherein the 3D image further includes a virtualrepresentation of a portion of the anatomy of the patient, and the 3Dimage is displayed in the field of view further such that the virtualrepresentation of the portion of the anatomy is collocated with theanatomy.
 31. The method of claim 29, further comprising: generating aset of images depicting different orthogonal or arbitrary planes of theportion of the anatomy; and displaying the set of images in the field ofview of the operator such that the set of images is shown next to the 3Dimage.
 32. The method of claim 31, wherein at least one of the set ofimages includes one or more additional guidance clues associated withthe suggested position or the suggested orientation of the medicaldevice.
 33. The method of claim 31, further comprising adapting the setsof images based on changes to the actual position and orientation of themedical device.
 34. The method of claim 29, further comprising:receiving, from the tracking system, tracking data associated with ahead of the operator; determining, based on the tracking data associatedwith the head of the operator, a position and orientation of the head ofthe operator; and adapting the 3D image based on the position andorientation of the head of the operator.